Cover page A Methodology for Diagnostic Evaluation of Spoken Human-Machine Dialogue Running head: Diagnostic Evaluation of Dialogue
نویسندگان
چکیده
Diagnostic evaluation is an important instrument for the development of high quality spoken language dialogue systems. Yet no rigorous methodology exists for the systematic and exhaustive diagnostic evaluation of all aspects of spoken language interaction: recognition, synthesis, grammar , vocabulary, dialogue etc. The paper addresses part of this problem by presenting a methodology for the diagnostic evaluation of spoken human-machine dialogue. The first part of the meth-3 odology is general and supports the detection of any kind of user-system interaction problem. The second part concerns the classification, diagnosis and repair of problems of dialogue interaction. These problems are of two broad kinds: dialogue design errors and user errors. Use of the methodology is demonstrated through analysis of a corpus from the user test of the Danish Dialogue System.
منابع مشابه
Generality and transferability. two issues in putting a dialogue evaluation tool into practical use
This paper presents a first set of test results on the generality and transferability of an evaluation tool which can ensure the habitability and usability of spoken dialogues. Building on the assumption that most, if not all, dialogue design errors can be viewed as problems of non-cooperative system behaviour, the tool has two closely related aspects to its use. Firstly, it may be used for the...
متن کاملPrinciples for the design of cooperative spoken human-machine dialogue
This paper presents a consolidated set of 24 principles of cooperative spoken human-machine dialogue which are based on the development and controlled user testing of the dialogue component of the Danish dialogue system as well as on comparison with human-human dialogue theory. Potentially, the principles could be used as effective and systematic dialogue development and evaluation tools both d...
متن کاملGENERALITY AND OBJECTIVITY Central Issues in Putting a Dialogue Evaluation Tool into Practical Use
This paper presents a first set of test results on the generality and objectivity of the Dialogue Evaluation Tool DET. Building on the assumption that most, if not all, dialogue design errors can be viewed as problems of non-cooperative system behaviour, DET has two closely related aspects to its use. Firstly, it may be used for the diagnostic evaluation of spoken human-machine dialogue. Follow...
متن کاملEvaluating the Effectiveness of Dialogue for an Automated Spoken Questionnaire
We present and apply an empirical methodology for evaluating the effectiveness of dialogues in spoken language systems. This methodology is suitable in particular for evaluation of dialogue-based systems that collect information from the user, such as an automated spoken questionnaire. Our method for assessing effectiveness involves coding answers from users for responsiveness. For this effort,...
متن کاملA Typology of Problems of Cooperativity in Spoken Human-machine Dialogue
The paper presents a method for identifying problems of usersystem interaction. Based on a consolidated set of 24 principles of cooperative spoken human-machine dialogue, the paper then proposes and illustrates a general typology of non-cooperative system dialogue behaviour for use in spoken language dialogue analysis and evaluation.
متن کامل